VADER: Video Alignment Differencing and Retrieval

We propose VADER, a spatio-temporal matching, alignment, and change summarization method to help fight misinformation spread via manipulated videos. VADER matches and coarsely aligns partial video fragments to candidate videos using a robust visual descriptor and scalable search over adaptively chunked video content. A transformer-based alignment module then refines the temporal localization of the query fragment within the matched video. A space-time comparator module identifies regions of manipulation between aligned content, invariant to any changes due to any residual temporal misalignments or artifacts arising from non-editorial changes of the content. Robustly matching video to a trusted source enables conclusions to be drawn on video provenance, enabling informed trust decisions on content encountered

Bias and Fairness in Large Language Models: A Survey

Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases. In this paper, we present a comprehensive survey of bias evaluation and mitigation techniques for LLMs. We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing, defining distinct facets of harm and introducing several desiderata to operationalize fairness for LLMs. We then unify the literature by proposing three intuitive taxonomies, two for bias evaluation, namely metrics and datasets, and one for mitigation. Our first taxonomy of metrics for bias evaluation disambiguates the relationship between metrics and evaluation datasets, and organizes metrics by the different levels at which they operate in a model: embeddings, probabilities, and generated text. Our second taxonomy of datasets for bias evaluation categorizes datasets by their structure as counterfactual inputs or prompts, and identifies the targeted harms and social groups; we also release a consolidation of publicly-available datasets for improved access. Our third taxonomy of techniques for bias mitigation classifies methods by their intervention during pre-processing, in-training, intra-processing, and post-processing, with granular subcategories that elucidate research trends. Finally, we identify open problems and challenges for future work. Synthesizing a wide range of recent research, we aim to provide a clear guide of the existing literature that empowers researchers and practitioners to better understand and prevent the propagation of bias in LLMs

Debiasing Image Generative Models

Generative models have become increasingly popular in various domains to solve challenging tasks, including image generation, dialogue generation, and story generation. Unlike discriminative models, they can learn the underlying probability distribution of data and generate new examples. In particular, image generative models have gained significant attention due to their remarkable ability to produce images of unparalleled quality. However, while there has been a lot of attention to biases in discriminative models, bias in generative models has received little attention. The presence of biases in generative models, particularly related to race and gender, can have significant consequences in downstream applications. Therefore, efforts to address this issue are essential to promote fair and ethical use of generative models in various domains. To achieve this goal, this dissertation presents a comprehensive study of debiasing image generative models by incorporating diversity and fairness constraints into the training process.In this dissertation, we investigate three different approaches to debiasing image generative models. In the first approach, a new task of high-fidelity image generation conditioned on multiple attributes from imbalanced datasets is proposed. This task poses new challenges for state-of-the-art GANs models, and a new training framework is proposed to address thesechallenges. The second approach investigates bias in image-to-image translation models and proposes debiasing using contrastive learning. Finally, the study highlights the prevalence of bias in large-pretrained models like CLIP and its impact on text-to-image generative models. Identity preserving losses are proposed to rectify the problem without retraining the pretrained model. In all of these approaches, we evaluate the impact of debiasing on image generation and the effectiveness of existing methods in reducing biases in generated images. We show the proposed task and framework offer new avenues for further research in debiasing generative models. Overall, this dissertation contributes to the field of generative models by providing a comprehensive study of debiasing generative models and proposing a new task and framework for high-fidelity image generation